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Abstract: Enhancing resource use efficiency in agricultural field management and breeding high-performance crop variet- 
ies are crucial approaches for securing crop yield and mitigating negative environmental impact of crop production. Crop 
stress sensing and plant phenotyping systems are integral to variable-rate (VR) field management and high-throughput 
plant phenotyping (HTPP), with both sharing similarities in hardware and data processing techniques. Crop stress sensing 
systems for VR field management have been studied for decades, aiming to establish more sustainable management prac- 
tices. Concurrently, significant advancements in HTPP system development have provided a technological foundation for 
reducing conventional phenotyping costs. In this paper, we present a systematic review of crop stress sensing systems em- 
ployed in VR field management, followed by an introduction to the sensors and data pipelines commonly used in field 
HTPP systems. State-of-the-art sensing and decision-making methodologies for irrigation scheduling, nitrogen application, 
and pesticide spraying are categorized based on the degree of modern sensor and model integration. We highlight the data 
processing pipelines of three ground-based field HTPP systems developed at the University of Nebraska-Lincoln. Further- 
more, we discuss current challenges and propose potential solutions for field HTPP research. Recent progress in artificial 
intelligence, robotic platforms, and innovative instruments is expected to significantly enhance system performance, en- 
couraging broader adoption by breeders. Direct quantification of major plant physiological processes may represent one of 
next research frontiers in field HTPP, offering valuable phenotypic data for crop breeding under increasingly unpredictable 
weather conditions. This review can offer a distinct perspective, benefiting both research communities in a novel manner. 
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1 Introduction pesticides plays an important role to close the yield 
gap™. Precision agriculture (PA) has been brought for- 
Securing food supply to meet the population i ; i 
; ; ward to realize a more sustainable crop production. 
growth and economic development under a changing 


; re sa Rather th lying irrigati ter, fertilizers, and 
climate is vital for the stability and development of a i aaa ae aaa a 
1:3] pesticides uniformly, these inputs are applied variably 


our society™™, The dominant contribution of precise 


field management and crop breeding to the increase of to match the spatial and temporal variability of crops 


the actual and potential crop yield is conclusive". Con- 
tinuously improving the utilization efficiency of agri- 


cultural inputs such as irrigation water, fertilizers, and 
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and soil within the field". Variable-rate (VR) applica- 
tion systems for irrigation, fertilization, and pesticides 


are carried out based on the soil and yield maps in the 
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USA". Numerous research projects have been con- 
ducted to quantify the real-time crop needs at high spa- 
tial and temporal resolutions to further improve the uti- 
lization efficiency of these input resources. VR field 
management involves three main phases, including 
raw data collection, application map generation, and 
application. Cameras with certain wavelength ranges 
have been widely used for raw data collection by sens- 
ing platforms like satellites, airplanes, drones, spray- 
ers, and field robots®'. Different platforms have pros 
and cons in data resolution and throughput, field cover- 
age, cost, and field accessibility. Other sensors like 
light detection and ranging (LiDAR) have also been 
used to measure canopy structure parameters related to 


its growth condition" 


. Models and algorithms have 
been developed to convert raw data to application 
maps. For example, the distribution map of the normal- 
ized difference vegetation index (NDVI) could be gen- 
erated by an aerial sensing platform following a stan- 
dard process. Then, an application map is generated 
based on an empirical model between the NDVI value 
and the application rate of nitrogen fertilizer"”. Simi- 
larly, other (a)biotic stresses can be quantified using 
appropriate sensing systems and converted to corre- 
sponding application maps for VR applications. The 
technologies for the application phase of VR field 
management are mature and commercially avail- 
able". For example, the irrigation map is converted 
into the operating parameters of center-pivots to trig- 
ger the irrigation event. Although huge effort has been 
made in the sensing and decision-making tools for VR 
field management, continuous research is still needed 
to develop tools that are reliable, versatile, well-inte- 
grated and affordable for wide adoption by producers. 
The optimal yield could not be achieved without 
continuously developing the desired seeds (e. g., cli- 
mate resilient, disease tolerant, and high nutrition con- 
centration). In the last decade, High-throughput plant 
phenotyping (HTPP), which focuses on selecting bet- 


ter-performance breeding lines in plant breeding using 


advanced sensing technologies, has made considerable 
progress by exploring suitable sensors, platforms, and 
data pipelines'*'*, Usually, the HTTP systems require 
higher spatial resolution for small breeding plots than 
that of VR field management at the production scale. 
The most widely used sensors in HTTP are cameras 
and LiDAR, which measure the canopy morphology, 
reflectance, and height at high throughput. Contact- 
type sensors are utilized for measuring leaf and soil pa- 
rameters at selected locations using in-field sensor net- 
works or robots!" Drones, high-clearance tractors, 
unmanned ground vehicles (UGVs), and large-scale 
ground systems have been developed for HTPP appli- 


24 These sensing and data analytic tools of 


cations! 
HTPP research could also be integrated into the raw 
data collection and the decision-making phases of VR 
field management. At the University of Nebraska-Lin- 
coln (UNL), several ground-based HTPP systems were 
developed for different research scenarios. 

This study aims to: 1) review the state-of-art sens- 
ing and decision-making technologies for VR field 
management; and 2) introduce the hardware and data 
processing pipelines of HTPP systems. We specifically 
examine crop stress sensing systems and their corre- 
sponding decision-making models for VR field man- 
agement, focusing on irrigation scheduling, nitrogen 
application, and pest management. Subsequently, we 
present the hardware and data pipelines of three HTPP 
systems developed at the UNL. Considering the simi- 
larities between the sensing systems in these two appli- 
cation areas, we hope this review can help researchers 
compare methodologies in both domains to further ad- 
vance the development of information systems for tar- 
geted applications. We also address the current bottle- 
necks hindering the wide adoption of HTPP systems 


and explore potential new research frontiers. 


2 Crop stress sensing systems for 
VR field management 


The main (a)biotic stresses in field crop produc- 
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tion include water, nutrient, and pests. A timely and ac- 
curate detection and quantification of these stresses is 
the first step to applying VR technologies. We divided 
this section into three specific topics and carried 
out the review work of the crop stress sensing sys- 
tems along with the decision-making models for 
Variable-Rate Irrigation (VRI) scheduling, nitrogen 
(N) application, and pesticide spray. 


2.1 Drought stress sensing and irrigation 
scheduling 


Irrigation for crop production often consumes the 
largest amount of fresh water, and the crop yield could 
be negatively impacted without irrigation”. There- 
fore, new ways to improve irrigation management and 
water use efficiency could effectively address the on- 
going water scarcity problem encountered by field 


M3" Table 1 lists the main sensing 


crop production 
technologies for irrigation scheduling to apply water 
with constant and variable depths. We list seven meth- 
ods from the experience-based one that does not re- 
quire sensor usage (I1) to data-driven methods that uti- 
lize sensors and models (I12—17). Feel and appearance 
(11) has been applied historically to estimate soil mois- 
ture with producers’ experience. Over or under irriga- 
tion often occurs using this method due to a lack of a 
quantitative measure of soil moisture. Spatial and tem- 
poral variation of soil water status within the field is 
one of the most important sources of information for 
VRI scheduling (12-17). Extensive research has fo- 
cused on investigating how this variation and the mi- 
croclimate relate to crop water stress. Thresholds of 
soil water potential were identified as irrigation trig- 
gers for specific soil textures and crop types (13) °”. 
Crop water stress index (CWSI) has been developed to 
quantify drought stress using infrared thermometers 


* Crop evapotranspira- 


and environmental data (14)! 
tion (ET) has been modeled using basic weather data 


and crop coefficients for the irrigation scheduling 


(I5)"". These data-driven methodologies have realized 
a more accurate guidance of the irrigation schedul- 
ing’. In other words, 12—I5 focus more on the irriga- 
tion scheduling to meet the real-time demand of crops. 
These methods often lack the needed spatial resolution 
to vary the water application rate site-specifically (i.e., 
only a few locations in the field can be monitored). 

16 and I7 are two more advanced methods based 
on a combination of a mobile sensing platform and sta- 
tionary sensor stations for VRI scheduling’”**. 16 le- 
verages the widely adopted center pivot irrigation sys- 
tems as an integrated sensing and irrigation platform 
by mounting infrared thermometers onboard to mea- 
sure the canopy temperature. An integrated CWSI was 
calculated to assess the spatial variation of crop water 
stress. Fully automatic VRI was realized by this meth- 
od without the additional cost of a standalone sensing 
platform. I7 deployed an aerial sensing platform to 
capture the spatial variation of the canopy reflectance. 
A hybrid model was developed based on surface ener- 
gy balance and soil water balance to model the spatial 
variation of the soil water dynamics. 

In conclusion, precise and site-specific irrigation 
management is essential for improving water use effi- 
ciency in field crop production, especially in regions 
where drought events are becoming increasingly fre- 
quent. Timely and accurate quantification of crop 
drought stress and corresponding water needs is a chal- 
lenging task, but it is necessary for effective irrigation 
scheduling. Our review highlights a range of irrigation 
scheduling methods, from traditional experience-based 
techniques to advanced data-driven approaches. In ad- 
dition to considering spatial variations in soil proper- 
ties and landscape, more advanced approaches utilize 
state-of-the-art remote sensing platforms and crop 
models to determine real-time crop water demand at 
high spatial resolutions. These data-driven methods 
provide a scientific foundation for moving towards ful- 


ly automatic VRI systems. 
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Table 1 Sensing methods for irrigation scheduling using variable-rate irrigation (VRI) systems 
Method Name Instrumentation Extracted parameters References 
Il Feel and appearance None Grower's observation [39 | 
1. Neutron probe 
2. Time Domain reflectometry 
; . 1. Allowable soil water depletion 
12 Soil water balance 3. Capacitance probes . . [36] 
. 2. Soil water depletion 
4. Tensiometers 
5. Granular matrix sensors 
13 Soil water potential Granular matrix sensors Soil water potential [32] 
i Crop water stress index . Infrared thermometer 1. Canopy temperature (33, 34] 
(CWSI) 2. Weather station 2. Weather data i 
Soil water balance with . 1. Volumetric water content at root zones 
. Soil water sensors 
I5 reference evapotranspira- . 2. Reference ET [35, 40] 
2. Weather station 
tion (ET) 3. Crop coefficient 
T . A wireless sensor network of Infrared 1. Canopy temperature 
> Integrated CWSI 
O 16 thermometers 2. Weather data [37, 41-43 ] 
O 2. Weather station 3. Volumetric water content at root zones 
N 1. Satellite imagery , 
, oe 1. Volumetric water content at root zones 
© Soil water balance 2. Aerial imagery 
(=) 17 2. Reference ET [21, 38, 44, 45] 
di with canopy reflectance 3. Weather station . 
3. Crop coefficient 
LO 4. Neutron probe 
A 2.2 Sensing technologies for nitrogen ap- agement (Table 2). 
(on) plication N deficiency often results in lower leaf chloro- 
N phyll content, which appears as reduced leaf green- 
High crop yield could not be achieved without ap- 
2 Š pa p ness. Leaf color charts are used as an affordable but ef- 
lying the appropriate amount of fertilizers to the : : SR” 
a p yng pprop fective way to guide the application of supplementary 
[46] ie : 
crops“. However, the loss of fertilizers to the environ- i 2 Fate : 
e p > N fertilizer (N1)"”. The application of optical chloro- 
ment through volatilization, surface runoff, and leach- : : : 
T 8 , á phyll sensors improved N use efficiency in the field 
ing ha n a major contributor to th radation of . ; . . 
(S) g has bee jor contributor to the degradation o experiment (N2)""!, Lab analysis of plant leaf tissues is 


the soil and water resources” *!, Potentially, excessive 
application of N fertilizer could lead to unsafe drink- 
ing water and food’: *', Adequate application of N fer- 
tilizer is vital for the high yield of the non-legume 
crop in most scenarios. The main challenge is to de- 
cide the right amount of application rate for high yield, 
good economical return, and minimum negative envi- 
ronmental impact. Previous research found that a uni- 
form N application could not achieve the best econom- 
ical return and led to N-related pollution in overfertil- 
ized areas within a field’. Therefore, the site-specific 
application rates of N fertilizer need to be tailored to 
what the crop needs. In this section, we reviewed sen- 


sors and scheduling methods developed for VR N man- 


considered the ground truth method to quantify plant 
N status (N3)™™. Typically, plant leaves are sampled 
and sent to a commercial lab for the measurement of 
total N concentration on a weight (g/g) or area (g/cm’) 
basis. However, this method is destructive, slow, labor 
intensive, with a low spatial-temporal resolution. Ma- 
chine Learning (ML) algorithms have been applied to 
realize high-throughput and non-destructive estima- 
tion of leaf macronutrients using field spectroscopy 
(N4)'". A higher spatial resolution could be achieved 
by integrating this technology into robotic systems". 
NS5—N7 are high-throughput sensing methods at the 
canopy level that provide the high spatial-temporal res- 


olution of the whole field for VR N application. A de- 
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Table 2 Sensing methods for variable-rate application of synthetic nitrogen fertilizer 

Method Name Instrumentation Extracted parameters References 

N1 Producer experience Leaf color chart Subjective leaf color [52] 

N2 Chlorophyll content SPAD chlorophyll meter Leaf chlorophyll content [53] 

N3 Lab analysis Nitrogen analyzer Leaf nitrogen concentration [20] 

N4 Leaf spectral scan VIS-NIR-SWIR spectrometer Leaf spectral reflectance [20] 

Active-optical reflectance i o = 
N5 : Active two-band NDVI sensor NDVI-related vegetation indices [55, 56, 60] 
sensor (AORS) algorithm 
. 1. NDVI-related vegetation indices 
1. Active three-band NDVI sensor . . o. 
: ae 2. Soil texture, apparent electrical conductivity, bulk 
N6 Extended AORS 2. Electrical conductivity sensor ; i [57] 
density, moisture, etc. 
3. Weather data . ooo 
3. Growing degree day, precipitation, etc. 

y= N7 Aerial platforms Hyperspectral camera Vegetation indices or reflectance spectrum [58, 59, 61] 
Cc tailed review of active and passive sensing methods In conclusion, achieving high crop yields in mod- 
= for this application was carried out"”. Active spectral ern agriculture necessitates the sustainable application 
O sensors have been developed specifically for this pur- of adequate nitrogen levels. Excessive nitrogen appli- 
> pose which measures the spatial distribution of selec- cation, however, can result in soil and water degrada- 
rts tive Vegetation indices (VIs). Sufficiency index (SI) is tion and potential human health issues. Our review 
ee) then calculated based on VIs of the reference strip or a covers a range of sensing technologies and scheduling 
© virtual reference (N5). The N recommendation rates methods designed to address this problem, from cost- 
N are then calculated from SI values using active-optical effective leaf color charts to sophisticated, integrated 
> reflectance sensor (AORS) algorithms". Outputs of sensing and application systems. We anticipate that the 
= 
x< active sensors are constant under varying illumination development of novel sensors and more generalized 
= conditions and could be used at night. These sensors models for variable-rate nitrogen scheduling will fur- 
Z are usually integrated into boom sprayers for real-time ther minimize the negative impacts of nitrogen overap- 
(S) VR N application. Economic and environmental bene- plication while maintaining high crop yields. 


fits were found by applying this method in field experi- 


ments"! 


. The spatial variation of soil properties and 
continuous weather data could further improve AORS 
performance (N6)'". Aerial sensing platforms (e. g., 
satellites, airplanes, and drones) provide a unique way 
to quantify the leaf chlorophyll content and biomass at 
the canopy level. Combined with ML algorithms, 
more complex models have been developed to esti- 
mate the plant N concentration (N7)""!. Early-season 
plant N uptake could be estimated using hyperspectral 
images in the field condition’. A study also shows 
that nitrogen use efficiency increases when using sens- 
ing systems in a non-irrigated maize field, compared 


to conventional application methods!” 


2.3 Biotic stress sensing for pesticide spray 


Widespread use of pesticides undoubtedly in- 
creases crop yield by protecting it from devastating 


pest damage” 


. However, the level of pesticide resi- 
dues entering food systems has to be strictly regulated 
by national agencies for human health while the intru- 
sion of pesticides into the environment has been moni- 
tored. Pesticide pollution often occurs from non-point 
sources like surface runoff, leaching and drainage, and 
spray drift. Pesticides were detected in surface and 
groundwater while higher concentrations are found in 


agricultural areas during the crop growing season'™®. 


Spray drift could be minimized by using drift-reduc- 
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tion nozzles under optimal weather conditions“, 


Among many mitigation strategies to reduce the pesti- 
cide presence in water bodies, reducing the total appli- 
cation volume could be one of the most effective and 
practical ways”, 

Imaging sensors, especially RGB camera, is the 
dominant tool to detect biotic stress, given that the ob- 
jects which cause the stress are often visible (weeds 
and insects). Table 3 summarizes the application of re- 
mote sensing methods and their processing algorithms 
to detect biotic stress for crop production. Commercial 
solutions for real-time spot-spray of weeds using 
boom sprayers are available from various vendors in 
recent years which could save up to 90% of herbicide 
usage (B1-B2'). Hyperspectral cameras have also 
been investigated for this task using ground platforms 


17 and airborne platforms at the sub- 


at the plant leve 
field level'!. Traditional ML models were trained to 
classify plant types using the features generated from 


"1 Then those selected features or 


the input images 
spectral reflectance are fed into a traditional machine 
learning (ML) model for classifying plant types. Usual- 
ly, the processing is not computationally intensive 
compared to deep learning (DL) methods. DL algo- 
rithms have been proven as an excellent tool to accu- 
rately discriminate weeds from crops under complex 
nature illumination". Light-weight field robots are an- 
other research focus to realize plant-level mechanical 
weeding to completely avoid herbicide needs (B3"”). 
Aerial platforms with high-resolution cameras and ad- 
vanced algorithms are potentially a universal sensing 
solution for field crop production in various scenari- 
os", B4 and B5 cover traditional ML and DL methods 
applied in the detection of crop disease and insect 
pests. Extensive effort has been made to improve the 
algorithm performance of the traditional image pro- 
cessing method under field conditions. An interdisci- 
plinary comparison was carried out for image process- 
ing techniques of the diagnosis of common human and 


plant diseases”. Public data sets of common plant dis- 


eases have been established to speed up the develop- 
ment of DL models®". Classification, detection and lo- 
calization, and segmentation DL neuron networks 
have been applied in plant disease diagnosis and insect 
detection *!, Labeled data sets, which at the target 
spatial resolution and cover the entire development of 
the plant disease (high temporal resolution) and a wide 
range of field conditions (e.g., illumination and man- 
agement practice), are the prerequisites to train more 
robust models”. Training models using cross-species 
data sets could also improve the model's perfor- 


mance!” 


. Commercial satellite imaging networks now 
can produce multispectral global images every day at 
sub-meter resolution (B6) and significantly relieves 
the resolution bottlenecks of using satellite sensing 
systems in VR field management. Agricultural internet 
of things (Ag-IoT) networks could collect images at 
sub-leaf resolution to directly detect crop disease and 
insect pests (B7). Furthermore, it could monitor gener- 
al crop growth by collecting crop-soil-environment da- 
ta at strategically selected locations in a field”, 

In conclusion, remote sensing technologies like 
RGB and hyperspectral imaging, when combined with 
ML and DL algorithms, have demonstrated potential 
in detecting biotic stress and therefore reducing pesti- 
cide use. Notably, DL networks have been developed 
to guide real-time chemical application, significantly 
decreasing herbicide consumption. By deploying suit- 
able platforms such as aerial platforms and field ro- 
bots, as well as developing more accurate and univer- 
sal models, we can effectively minimize crop biotic 


stress while preserving the environment. 


3 Ground-based field HTPP sys- 
tems at UNL 


HTPP systems need to address various questions 
depending on the breeder's needs. Generally, they are 
used to help breeders to select better genotypes at spe- 
cific breeding stages. Field HTPP systems usually uti- 


lize sensors, which have been seen in VR field man- 
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Table 3 A case summary in biotic stress detection for crop production 

Method Target stress Name Sensing instruments Algorithm References 
Bl Weeds John Deer's See & Spray RGB camera Deep learning Vendor website 

Trimble Agriculture's Weed- . ; 
B2 Weeds Active NDVI sensor NDVI threshold Vendor website 

seeker 2 

Small Robot Company's Tom . S 

B3 Weeds RGB camera Deep learning Vendor website 


and Dick 


RGB, multispectral, hyperspectral, 


B4 Disease and insects Traditional image processing Traditional and deep learning [80, 86] 
thermal cameras 
, : Deep learning—based diag- RGB camera, . 

B5 Disease and insects Deep learning [81-83, 87] 

nose Hyperspectral camera 
High-resolution satellite net- f . . 

B6 General crop status i Multispectral camera Traditional and deep learning [23, 88] 
wor 

B7 General crop status Ag IoT network Various types of sensors Traditional and deep learning [85, 89] 


agement, to quantify crop parameters that are impor- 
tant to crop breeders. Although similar sensor clusters 
can be used in both areas, HTPP systems generally re- 
quire high spatial resolution and accuracy due to the 
small footprints of the breeding plot. 

Distinct combinations of sensors and platforms 
yield unique advantages and disadvantages for HTPP 
systems. Generally, most HTPP systems utilize camer- 
as and other sensors at varying spatial resolutions to 
measure similar phenotypic traits, such as leaf color 
and texture properties, canopy height and structure, 
canopy coverage rate, spectral reflectance, and canopy 
temperature. In current HTPP data pipelines, these 
traits are predominantly employed as selection criteria 
for crop breeding. Given the similarities in data for- 
mats and processing pipelines among HTPP systems, 
we present a comprehensive review of the primary sen- 
sors and their corresponding data pipelines used in 
HTPP research, illustrated through a detailed examina- 
tion of three HTPP systems developed at UNL. 

Fig. 1 shows these platforms and corresponding 
sensors. Various phenotypic traits could be collected 
by them, such as canopy coverage rate, temperature, 
spectral reflectance, and height. The greenhouse phe- 
notyping system (Fig. 1(A)) captures multispectral 


and thermal images for experiment plots. The pheno- 
cart (Fig. 1(B)) can measure over 1500 breeding plots 
at a breeding location in one day. NU-Spidercam 
(Fig. 1(C)) is a large-scale cable-suspend field pheno- 
typing facility with unique advantages. Main onboard 
sensors for each system are indicated in the corre- 


sponding sub-figures. 
3.1 Multispectral-thermal platform 


3.1.1 System components and raw data 

The greenhouse phenotyping system was used 
to quantify the difference in drought dress of winter 
wheat plants under different irrigation treatments 
(Table 4)". Canopy growth was quantified by calculat- 
ing CWSI and growth index (GI) for each plot based 
on the multispectral and thermal images. 
3.1.2 Data processing and data output 

The experiment was conducted in a greenhouse 
with bare soil. The wheat genotypes with different 
drought stress responses were planted in 28 plots with 


1 The plots were evenly split 


a row spacing of 18 cm 
into drought and well-irrigated groups and raw images 
were collected multiple days across the growing peri- 
od. Data pre-processing was carried out before the in- 


dex calculation (Fig. 2). Image segmentation and regis- 
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NDVI sensor 
Air temperature 
Relative humidity 


Pan and Tilt system 


RGB camera 
Ultrasonic sensor 
NDVI sensor 
Spectrometer system 
Infrared thermometer 


(B) 


Multispectral camera 
Thermal infrared camera 
Spectrometer 

LiDAR 

Hyperspectral camera 


© 


Note: (A) a mobile platform for sensing drought stress; (B) a field phenocart for measuring various phenotypic traits; (C) a large- 


scale cable-suspended field phenotyping facility (NU-Spidercam ) 


Fig. 1 Ground-based HTPP systems developed at UNL 


Table 4 Instrumentation of the greenhouse phenotyping cart 


Camera type Camera info 


Other Parameters 


Thermal camera 


NIR monochrome camera 


6-band filter wheel 


SC640, FLIR, OR, USA 


640 X 480 pixels 
Canopy temperature 


1280 X 1024 pixels 
Canopy coverage 


6 bands: 530, 570, 670, 770, 870, and 970 nm 
Canopy coverage and NDVI reflectance 


DCC3240N, Thorlabs, NJ, USA 


FW102C, Thorlabs, NJ, USA 


tration were applied to extract the canopy coverage 
rate and the average leaf temperature. Air temperature 
and relative humidity were recorded by a standalone 
sensor station at the center of the experiment area. GI 
values were calculated by multiplying the canopy 
height and coverage rate. CWSI was calculated using 
the canopy and environmental parameters. 
3.1.3 Result summary 

The result shows that CWSI could differentiate 
the drought stresses between the given water treat- 
ments while the canopy coverage rate and GI could 
quantify the general plant vigor based on the correla- 
tion analysis between active NDVI sensors and the 


two parameters’. The system proves that a simple 


combination of two cameras could be used to identify 
drought-tolerant genotypes based on these phenotypic 
measures. More phenotypic parameters could be ob- 
tained by integrating more sensors. For example, the 
RGB-Depth camera could be used for canopy height 
measurement and coverage rate; Canopy structural pa- 
rameters could be measured by LiDAR; Spectral re- 
flectance of the canopy could be quantified using the 
hyperspectral camera. Little wind disturbance is an im- 
portant advantage in the greenhouse, which has made 
important image processing processes (e.g., image reg- 


istration) more accurate. 
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Red 


NIR 


Thermal 


Registered thermal image 


Environmental data 


* Average plant height 
* Canopy coverage rate 
* Average leaf temperature 


{ 


* Canopy Water Stress Index (CWSI) 
* Growth Index (GI) 


Fig. 2 The image processing protocol to calculate growth index (GI) and canopy water stress index (CWSI) by processing 


multispectral and thermal images with the environmental data 


3.2 Integrated sensing platform-Phenocart 


3.2.1 System components and raw data 
A more complex phenotyping system was devel- 


oped for HTPP under the field condition for multiple 


91,92 


crop-site-year trials”'””, Table 5 lists the onboard sen- 
sor types, detailed sensor information, and the extract- 
ed phenotypic parameters of the system. Multiple carts 
were built to support field breeding experiments of 


maize, soybean, winter wheat, and other crops. 


Table 5 Instrumentation of Phenocart and corresponding phenotypic parameters 


Sensor type 


Sensor info 


Other Parameters 


RGB camera 


Ultrasonic sensor 
Thermal radiometer 
NDVI sensor 
Spectrometer 


Air temperature and relative humidity sensor 


GPS AgGPS 162, Trimble Agriculture, CA, USA 


ToughSonic30, Senix Corporation, VT, USA. 
SI-131, Apogee Instruments, Logan, UT, USA. 
SRS, Meter Group, WA, USA. 
CCS175, Thorlabs, NJ, USA 
HMP4SC-L, Campbell Sci., UT, USA 


1920 X 1080 pixels 
Canopy coverage 


C615, Logitech, Lausanne, Switzerland 


Canopy height 
Plot temperature 
Plot NDVI 
Plot reflectance 
Air temperature and relative humidity 


Location and Timestamp 


3.2.2 Data processing and data output 
Raw RGB images for every breeding plot were 


stored in the root directory with a unique file name 
that indicates the plot number, experiment name, and 
timestamp. All other point measurements were record- 
ed in a single data table in which each row represents 
the raw data of a point measurement. These data in- 
clude the plot number, GPS coordinates, timestamps, 


height of the ultrasonic sensor, distance data from the 


ultrasonic sensor, plot temperature from the thermal ra- 
diometer, plot NDVI, plot reflectance, air temperature, 
and relative humidity. Fig. 3 illustrates the image pro- 
cessing pipeline for the automatic scoring of iron defi- 
ciency chlorosis (IDC) in a soybean breeding experi- 
ment””!, The extracted color features using the protocol 
are used as inputs of statistical models to realize the 


ML-based IDC scoring. 
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Divide into sections 


Green Ratio 


Detect the crop line => 


for each section Le oy ees x 


Yellow Ratio 


Exclude the weeds 


Brown Ratio 


Fig. 3 The image processing protocol of automatic grading for soybean plots with stress from iron deficiency chlorosis (IDC) 


3.2.3 Result summary 

The multi-year data collection proves that pheno- 
cart is a reliable and relatively affordable HTPP tool 
for crop breeders to collect various phenotypic parame- 


P41 The hardware and control software worked 


ters 
seamlessly and interactively with operators in the field 
condition. A reasonably high speed of about 0.2 ha/h 
was achieved by two operators manually pushing or 
pulling the cart in the field. The cart was also operated 
throughout the growing season for short-stature crops 
like winter wheat and camelina. The narrow tire width 
and protection cover minimized the plant damage dur- 
ing the data collection. The spatial resolution of the 
RGB images was excellent because of the close dis- 
tance between the cameras and the canopy. The ther- 
mal radiometer measured the canopy temperature accu- 
rately although it could not distinguish the plant and 
soil targets within its field of view (FoV). In the later 
version of the phenocart, we replaced the ultrasonic 
sensors with a single LiDAR with 180° FoV perpen- 
dicular to the direction of the crop row. The LiDAR 
measured the canopy height more accurately than the 
ultrasonic sensor in a winter wheat experiment”. A 
good application of the phenocart was the automatic 


"l By imaging soybean 


rating of soybean IDC stress 
plots under IDC stress, a ML-based approach was tak- 
en to automatically score the IDC symptom (Fig. 3). 
The high prediction accuracy shows a great potential 


to integrate this work into breeding projects to im- 


prove screening efficiency. 


3.3 Large-scale, cable-suspend phenotyp- 
ing facility-Spidercam 


3.3.1 System components and raw data 

NU-Spidercam was built in 2017 at the Eastern 
Nebraska Research, Extension, and Education Center 
at UNL”. This cable-suspended sensing platform 
moves within a 0.4 ha scanning field precisely. In addi- 
tion to environmental sensors, GPS, and other electron- 
ics, most of the sensors are mounted on a rotation 
frame which provides pitch and yaw flexibility. Table 
6 lists the main instrumentation integrated into the NU- 
Spidercam. The raw data set includes multispectral 
images, thermal images, hyperspectral images, LIDAR 
point clouds, and the spectral reflectance. Frequent 
scanning of the whole field was carried out during the 
growing season. On each day, multiple measurements 
can be taken for each plot to track the diurnal dynam- 
ics of highly-variable phenotypic parameters. 
3.3.2 Data processing and data output 

All the raw data of one plot at a specific time was 
saved in a folder with a unique folder name. The fold- 
er name includes essential information including the 
time stamp, the plot number, the platform location, 
and sensor angles. Several phenotypic parameters 
were generated during the data processing. Fig. 4 
gives a typical data processing pipeline for the multi- 


spectral camera, thermal camera, spectrometer, and the 
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Table 6 Instrumentation of NU-Spidercam HTPP facility 


Sensor type 


Sensor info 


Other Parameters 


Multispectral camera 


Thermal camera 


Spectrometer 


LiDAR 


Hyperspectral camera 


ADO80GE, JAI, Miyazaki, Japan 


HR2000+, Ocean Insight, FL, USA 


VLP-16 Puck, Velodyne, CA, USA 


1024 X 768 pixel 
Canopy coverage 


640 X 480 pixels 
Canopy and soil temperature 


A655sc, Teledyne FLIR, OR, USA 


Plot reflectance 


Canopy height and structure 
362— 1043 nm 


HSV101, Middleton, WI, USA 


Canopy reflectance 


LiDAR. The canopy coverage rate was obtained by 
segmenting the plant pixels from the soil background 
in the multispectral images. Image segmentation and 
registration were applied to calculate the average soil 
and canopy temperature using the thermal image and 
the binary mask from the segmentation. NDVI, photo- 
chemical reflectance index (PRI), Red-Edge NDVI, 
soil-adjusted vegetation index (SAVT), and other VIs 


Thermal 
Registration 


* Spectral reflectance 


* LiDAR point cloud 


aero 


d Se 


Spectrometer calibration and 
Point cloud processing 


were calculated from the spectral reflectance from the 
spectrometer and hyperspectral camera. Canopy height 
was calculated from the LiDAR point cloud by sub- 
tracting the distance between the sensor and the cano- 
py top from the distance between the ground and the 
sensor. All the parameters were output as a single da- 
ta table with matched weather data which is delivered 


to the facility users. 


Canopy coverage 
Canopy temperature 


Soil temperature 
Canopy NDVI 


a cats 


‘gmentation 


Reflectance spectrum 
Vegetation indices 


Canopy height 


Fig. 4 Data processing pipelines for the onboard sensor cluster, including the multispectral camera, thermal camera, spectrometer , 


and the LIDAR 


3.3.3 Result summary 


NU-Spidercam has been successfully operated 
for 6 years without major issues. Effective wind pro- 
tection structures were added to avoid cable wrapping 
on the poles due to the frequent strong wind at the fa- 
cility site. Data collection can last for 6—8 hours with- 
out battery replacement with hot-swap capacity for ex- 


tended collection duration. The performance of the 


platform positioning is accurate enough for the small 


plot study with an estimated accuracy of +10 cm, giv- 
en a plot size of 4.6 m by 6.1 m at the site. No crop 
damage was created by the sensing platform and the 
shadow of the platform was minimized. The platform 
can be raised up to 10 m above the ground which en- 
ables us to scan tall crops (e. g., energy sorghum) 


while keeping a wide sensor FoV. 
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4 Conclusion 


The utilization of VR field management for irriga- 
tion water, fertilizers, and pesticides holds great prom- 
ise in enhancing crop yield, optimizing resource use, 
and reducing adverse environmental consequences re- 
sulting from over-application. The increasing preva- 
lence of climate change has led to heightened interest 
in sensing systems and their corresponding decision- 
making models. Concurrently, significant strides have 
been made in HTPP research, with the development of 
integrated sensing platforms and data pipelines for 
crop breeding aimed at bolstering phenotyping effi- 
ciency. Compared to VR field management, HTPP sys- 
tems necessitate the detection of plant stress at a much 
finer spatial resolution. This review uniquely explores 
the sensing systems employed in both VR field man- 
agement and HTPP systems. We initially examined 
crop stress sensing systems and their data pipelines for 
field crop production, concentrating on irrigation 
scheduling, nitrogen application, and pesticide spray- 
ing. Subsequently, we discussed the hardware and data 
pipeline of field HTPP systems by presenting three 
ground-based systems developed at UNL. 

Significant advancements in field HTPP research 
for crop breeding have resulted from the interdisciplin- 
ary collaboration. It is anticipated that the integration 
of these platforms and data pipelines into crop breed- 
ing programs will enhance phenotyping efficiency, pri- 
marily by reducing labor costs and generating high- 
quality data sets. However, further improvements in 
their performance and cost are required to encourage 
widespread adoption. Overcoming challenges related 
to data quality, collection throughput, and processing 
pipeline efficiency is crucial for HTPP system optimi- 
zation. The recent developments in artificial intelli- 
gence, robotic platforms, and innovative sensors are 
poised to enhance system performance and better sup- 
port crop breeding decisions. 

Optical sensors, particularly cameras capturing 


images at various wavelengths, are central to field 


HTPP systems due to their high-throughput capacity 
and spatial resolution. Conventional and DL-based 
computer vision algorithms allow for the accurate ex- 
traction of morphological traits from target canopies or 
individual plants as crucial phenotypic traits. These 
morphological traits are the cumulative product of ma- 
jor crop physiological processes such as ET, photosyn- 
thesis, and respiration. Developing phenotyping tools 
that can more directly quantify the physiological pro- 
cesses at the plot and plant level (real-time physiology 
phenotyping) will offer valuable insights for breeding 
climate-resilient varieties. Specifically, adapting sens- 
ing and modeling technologies from the remote sens- 
ing community to HTPP, with the support of high-reso- 
lution data sets, could facilitate a more direct estima- 
tion of these critical physiological processes (e.g., ET 
models and measurement of solar-induced fluores- 


cence). 
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